LSST Applications  21.0.0-172-gfb10e10a+18fedfabac,22.0.0+297cba6710,22.0.0+80564b0ff1,22.0.0+8d77f4f51a,22.0.0+a28f4c53b1,22.0.0+dcf3732eb2,22.0.1-1-g7d6de66+2a20fdde0d,22.0.1-1-g8e32f31+297cba6710,22.0.1-1-geca5380+7fa3b7d9b6,22.0.1-12-g44dc1dc+2a20fdde0d,22.0.1-15-g6a90155+515f58c32b,22.0.1-16-g9282f48+790f5f2caa,22.0.1-2-g92698f7+dcf3732eb2,22.0.1-2-ga9b0f51+7fa3b7d9b6,22.0.1-2-gd1925c9+bf4f0e694f,22.0.1-24-g1ad7a390+a9625a72a8,22.0.1-25-g5bf6245+3ad8ecd50b,22.0.1-25-gb120d7b+8b5510f75f,22.0.1-27-g97737f7+2a20fdde0d,22.0.1-32-gf62ce7b1+aa4237961e,22.0.1-4-g0b3f228+2a20fdde0d,22.0.1-4-g243d05b+871c1b8305,22.0.1-4-g3a563be+32dcf1063f,22.0.1-4-g44f2e3d+9e4ab0f4fa,22.0.1-42-gca6935d93+ba5e5ca3eb,22.0.1-5-g15c806e+85460ae5f3,22.0.1-5-g58711c4+611d128589,22.0.1-5-g75bb458+99c117b92f,22.0.1-6-g1c63a23+7fa3b7d9b6,22.0.1-6-g50866e6+84ff5a128b,22.0.1-6-g8d3140d+720564cf76,22.0.1-6-gd805d02+cc5644f571,22.0.1-8-ge5750ce+85460ae5f3,master-g6e05de7fdc+babf819c66,master-g99da0e417a+8d77f4f51a,w.2021.48
LSST Data Management Base Package
Public Member Functions | Static Public Member Functions | Static Public Attributes | List of all members
lsst.obs.decam.ingestCalibs.DecamCalibsParseTask Class Reference
Inheritance diagram for lsst.obs.decam.ingestCalibs.DecamCalibsParseTask:
lsst.pipe.tasks.ingestCalibs.CalibsParseTask lsst.pipe.tasks.ingest.ParseTask

Public Member Functions

def getInfo (self, filename)
 
def translate_ccdnum (self, md)
 
def translate_date (self, md)
 
def translate_filter (self, md)
 
def getDestination (self, butler, info, filename)
 
def getCalibType (self, filename)
 
def getInfoFromMetadata (self, md, info=None)
 

Static Public Member Functions

def getExtensionName (md)
 

Static Public Attributes

 ConfigClass = ParseConfig
 
 translator_class = None
 

Detailed Description

Parse calibration products for ingestion.

Handle either DECam Community Pipeline calibration products or
calibration products produced with the LSST Science Pipelines
(i.e., pipe_drivers' constructBias.py and constructFlat.py).

Definition at line 10 of file ingestCalibs.py.

Member Function Documentation

◆ getCalibType()

def lsst.pipe.tasks.ingestCalibs.CalibsParseTask.getCalibType (   self,
  filename 
)
inherited 
Return a a known calibration dataset type using
the observation type in the header keyword OBSTYPE

@param filename: Input filename

Definition at line 21 of file ingestCalibs.py.

21  def getCalibType(self, filename):
22  """Return a a known calibration dataset type using
23  the observation type in the header keyword OBSTYPE
24 
25  @param filename: Input filename
26  """
27  md = readMetadata(filename, self.config.hdu)
28  if not md.exists("OBSTYPE"):
29  raise RuntimeError("Unable to find the required header keyword OBSTYPE in %s, hdu %d" %
30  (filename, self.config.hdu))
31  obstype = md.getScalar("OBSTYPE").strip().lower()
32  if "flat" in obstype:
33  obstype = "flat"
34  elif "zero" in obstype or "bias" in obstype:
35  obstype = "bias"
36  elif "dark" in obstype:
37  obstype = "dark"
38  elif "fringe" in obstype:
39  obstype = "fringe"
40  elif "sky" in obstype:
41  obstype = "sky"
42  elif "illumcor" in obstype:
43  obstype = "illumcor"
44  elif "defects" in obstype:
45  obstype = "defects"
46  elif "qe_curve" in obstype:
47  obstype = "qe_curve"
48  elif "linearizer" in obstype:
49  obstype = "linearizer"
50  elif "crosstalk" in obstype:
51  obstype = "crosstalk"
52  elif "BFK" in obstype:
53  obstype = "bfk"
54  elif "photodiode" in obstype:
55  obstype = 'photodiode'
56  return obstype
57 
strip
bool strip
Definition: fits.cc:911
lsst::afw::fits::readMetadata
std::shared_ptr< daf::base::PropertyList > readMetadata(std::string const &fileName, int hdu=DEFAULT_HDU, bool strip=false)
Read FITS header.
Definition: fits.cc:1657

◆ getDestination()

def lsst.obs.decam.ingestCalibs.DecamCalibsParseTask.getDestination (   self,
  butler,
  info,
  filename 
) 
Get destination for the file.

Parameters
----------
butler : `lsst.daf.persistence.Butler`
    Data butler.
info : data ID
    File properties, used as dataId for the butler.
filename : `str`
    Input filename.

Returns
-------
raw : `str`
    Destination filename.

Reimplemented from lsst.pipe.tasks.ingestCalibs.CalibsParseTask.

Definition at line 139 of file ingestCalibs.py.

139  def getDestination(self, butler, info, filename):
140  """Get destination for the file.
141 
142  Parameters
143  ----------
144  butler : `lsst.daf.persistence.Butler`
145  Data butler.
146  info : data ID
147  File properties, used as dataId for the butler.
148  filename : `str`
149  Input filename.
150 
151  Returns
152  -------
153  raw : `str`
154  Destination filename.
155  """
156  calibType = self.getCalibType(filename)
157  md = readMetadata(filename, self.config.hdu)
158  if "PROCTYPE" not in md:
159  raise RuntimeError(f"Unable to find the calib header keyword PROCTYPE \
160  in {filename}, hdu {self.config.hdu}")
161  proctype = md["PROCTYPE"].strip()
162  if "MasterCal" in proctype: # DECam Community Pipeline calibration product case
163  # Arbitrarily set ccdnum and calib_hdu to 1 to make the mapper template happy
164  info["ccdnum"] = 1
165  info["calib_hdu"] = 1
166  if "flat" in calibType.lower():
167  raw = butler.get("cpFlat_filename", info)[0]
168  elif ("bias" or "zero") in calibType.lower():
169  raw = butler.get("cpBias_filename", info)[0]
170  elif ("illumcor") in calibType.lower():
171  raw = butler.get("cpIllumcor_filename", info)[0]
172  else:
173  raise RuntimeError(f"Invalid DECam Community Pipeline calibType {calibType}")
174  else: # LSST-built calibration product case
175  raw = butler.get(calibType + "_filename", info)[0]
176  # Remove HDU extension (ccdnum) since we want to refer to the whole file
177  c = raw.find("[")
178  if c > 0:
179  raw = raw[:c]
180  return raw

◆ getExtensionName()

def lsst.pipe.tasks.ingest.ParseTask.getExtensionName (   md)
staticinherited 
 Get the name of a FITS extension.

Parameters
----------
md : `lsst.daf.base.PropertySet`
    FITS header metadata.

Returns
-------
result : `str` or `None`
    The string from the EXTNAME header card if it exists. None otherwise.

Definition at line 126 of file ingest.py.

126  def getExtensionName(md):
127  """ Get the name of a FITS extension.
128 
129  Parameters
130  ----------
131  md : `lsst.daf.base.PropertySet`
132  FITS header metadata.
133 
134  Returns
135  -------
136  result : `str` or `None`
137  The string from the EXTNAME header card if it exists. None otherwise.
138  """
139  try:
140  ext = md["EXTNAME"]
141  except KeyError:
142  ext = None
143  return ext
144 

◆ getInfo()

def lsst.obs.decam.ingestCalibs.DecamCalibsParseTask.getInfo (   self,
  filename 
) 
Retrieve path, calib_hdu, and possibly calibDate.

Parameters
----------
filename: `str`
    Calibration file to inspect.

Returns
-------
phuInfo : `dict`
    Primary header unit info.
infoList : `list` of `dict`
    List of file properties to use for each extension.

Reimplemented from lsst.pipe.tasks.ingest.ParseTask.

Definition at line 18 of file ingestCalibs.py.

18  def getInfo(self, filename):
19  """Retrieve path, calib_hdu, and possibly calibDate.
20 
21  Parameters
22  ----------
23  filename: `str`
24  Calibration file to inspect.
25 
26  Returns
27  -------
28  phuInfo : `dict`
29  Primary header unit info.
30  infoList : `list` of `dict`
31  List of file properties to use for each extension.
32  """
33  phuInfo, infoList = CalibsParseTask.getInfo(self, filename)
34  # In practice, this returns an empty dict
35  # and a list containing an empty dict.
36  for item in infoList:
37  item['path'] = filename
38  try:
39  item['calib_hdu'] = item['hdu']
40  except KeyError: # calib_hdu is required for the calib registry
41  item['calib_hdu'] = 1
42  # Try to fetch a date from filename
43  # and use as the calibration dates if not already set
44  found = re.search(r'(\d\d\d\d-\d\d-\d\d)', filename)
45  if found:
46  date = found.group(1)
47  for info in infoList:
48  if 'calibDate' not in info or info['calibDate'] == "unknown":
49  info['calibDate'] = date
50  return phuInfo, infoList
51 

◆ getInfoFromMetadata()

def lsst.pipe.tasks.ingest.ParseTask.getInfoFromMetadata (   self,
  md,
  info = None 
)
inherited 
Attempt to pull the desired information out of the header

This is done through two mechanisms:
* translation: a property is set directly from the relevant header keyword
* translator: a property is set with the result of calling a method

The translator methods receive the header metadata and should return the
appropriate value, or None if the value cannot be determined.

@param md      FITS header
@param info    File properties, to be supplemented
@return info

Definition at line 145 of file ingest.py.

145  def getInfoFromMetadata(self, md, info=None):
146  """Attempt to pull the desired information out of the header
147 
148  This is done through two mechanisms:
149  * translation: a property is set directly from the relevant header keyword
150  * translator: a property is set with the result of calling a method
151 
152  The translator methods receive the header metadata and should return the
153  appropriate value, or None if the value cannot be determined.
154 
155  @param md FITS header
156  @param info File properties, to be supplemented
157  @return info
158  """
159  if info is None:
160  info = {}
161  for p, h in self.config.translation.items():
162  value = md.get(h, None)
163  if value is not None:
164  if isinstance(value, str):
165  value = value.strip()
166  info[p] = value
167  elif p in self.config.defaults:
168  info[p] = self.config.defaults[p]
169  else:
170  self.log.warning("Unable to find value for %s (derived from %s)", p, h)
171  for p, t in self.config.translators.items():
172  func = getattr(self, t)
173  try:
174  value = func(md)
175  except Exception as e:
176  self.log.warning("%s failed to translate %s: %s", t, p, e)
177  value = None
178  if value is not None:
179  info[p] = value
180  return info
181 

◆ translate_ccdnum()

def lsst.obs.decam.ingestCalibs.DecamCalibsParseTask.translate_ccdnum (   self,
  md 
) 
Return CCDNUM as a integer.

Parameters
----------
md : `lsst.daf.base.PropertySet`
    FITS header metadata.

Definition at line 62 of file ingestCalibs.py.

62  def translate_ccdnum(self, md):
63  """Return CCDNUM as a integer.
64 
65  Parameters
66  ----------
67  md : `lsst.daf.base.PropertySet`
68  FITS header metadata.
69  """
70  if "CCDNUM" in md:
71  ccdnum = md["CCDNUM"]
72  else:
73  return self._translateFromCalibId("ccdnum", md)
74  # Some MasterCal from NOAO Archive has 2 CCDNUM keys in each HDU
75  # Make sure only one integer is returned.
76  if isinstance(ccdnum, collections.abc.Sequence):
77  try:
78  ccdnum = ccdnum[0]
79  except IndexError:
80  ccdnum = None
81  return ccdnum
82 

◆ translate_date()

def lsst.obs.decam.ingestCalibs.DecamCalibsParseTask.translate_date (   self,
  md 
) 
Extract the date as a strong in format YYYY-MM-DD from the FITS header DATE-OBS.
Return "unknown" if the value cannot be found or converted.

Parameters
----------
md : `lsst.daf.base.PropertySet`
    FITS header metadata.

Reimplemented from lsst.pipe.tasks.ingest.ParseTask.

Definition at line 83 of file ingestCalibs.py.

83  def translate_date(self, md):
84  """Extract the date as a strong in format YYYY-MM-DD from the FITS header DATE-OBS.
85  Return "unknown" if the value cannot be found or converted.
86 
87  Parameters
88  ----------
89  md : `lsst.daf.base.PropertySet`
90  FITS header metadata.
91  """
92  if "DATE-OBS" in md:
93  date = md["DATE-OBS"]
94  found = re.search(r'(\d\d\d\d-\d\d-\d\d)', date)
95  if found:
96  date = found.group(1)
97  else:
98  self.log.warn("DATE-OBS does not match format YYYY-MM-DD")
99  date = "unknown"
100  elif "CALIB_ID" in md:
101  date = self._translateFromCalibId("calibDate", md)
102  else:
103  date = "unknown"
104  return date
105 

◆ translate_filter()

def lsst.obs.decam.ingestCalibs.DecamCalibsParseTask.translate_filter (   self,
  md 
) 
Extract the filter name.

Translate a full filter description into a mere filter name.
Return "unknown" if the keyword FILTER does not exist in the header,
which can happen for some valid Community Pipeline products.

Parameters
----------
md : `lsst.daf.base.PropertySet`
    FITS header metadata.

Returns
-------
filter : `str`
    The name of the filter to use in the calib registry.

Reimplemented from lsst.pipe.tasks.ingest.ParseTask.

Definition at line 106 of file ingestCalibs.py.

106  def translate_filter(self, md):
107  """Extract the filter name.
108 
109  Translate a full filter description into a mere filter name.
110  Return "unknown" if the keyword FILTER does not exist in the header,
111  which can happen for some valid Community Pipeline products.
112 
113  Parameters
114  ----------
115  md : `lsst.daf.base.PropertySet`
116  FITS header metadata.
117 
118  Returns
119  -------
120  filter : `str`
121  The name of the filter to use in the calib registry.
122  """
123  if "FILTER" in md:
124  if "OBSTYPE" in md:
125  obstype = md["OBSTYPE"].strip().lower()
126  if "zero" in obstype or "bias" in obstype:
127  return "NONE"
128  filterName = CalibsParseTask.translate_filter(self, md)
129  # TODO (DM-24514): remove workaround if/else
130  if filterName == '_unknown_' and "CALIB_ID" in md:
131  return self._translateFromCalibId("filter", md)
132  else:
133  return CalibsParseTask.translate_filter(self, md)
134  elif "CALIB_ID" in md:
135  return self._translateFromCalibId("filter", md)
136  else:
137  return "unknown"
138 

Member Data Documentation

◆ ConfigClass

lsst.pipe.tasks.ingest.ParseTask.ConfigClass = ParseConfig
staticinherited

Definition at line 70 of file ingest.py.

◆ translator_class

lsst.pipe.tasks.ingest.ParseTask.translator_class = None
staticinherited

Definition at line 71 of file ingest.py.

The documentation for this class was generated from the following file:
Generated on Fri Nov 26 2021 10:16:26 for LSST Applications by doxygen 1.9.1