LSST Applications  21.0.0+04719a4bac,21.0.0-1-ga51b5d4+f5e6047307,21.0.0-11-g2b59f77+a9c1acf22d,21.0.0-11-ga42c5b2+86977b0b17,21.0.0-12-gf4ce030+76814010d2,21.0.0-13-g1721dae+760e7a6536,21.0.0-13-g3a573fe+768d78a30a,21.0.0-15-g5a7caf0+f21cbc5713,21.0.0-16-g0fb55c1+b60e2d390c,21.0.0-19-g4cded4ca+71a93a33c0,21.0.0-2-g103fe59+bb20972958,21.0.0-2-g45278ab+04719a4bac,21.0.0-2-g5242d73+3ad5d60fb1,21.0.0-2-g7f82c8f+8babb168e8,21.0.0-2-g8f08a60+06509c8b61,21.0.0-2-g8faa9b5+616205b9df,21.0.0-2-ga326454+8babb168e8,21.0.0-2-gde069b7+5e4aea9c2f,21.0.0-2-gecfae73+1d3a86e577,21.0.0-2-gfc62afb+3ad5d60fb1,21.0.0-25-g1d57be3cd+e73869a214,21.0.0-3-g357aad2+ed88757d29,21.0.0-3-g4a4ce7f+3ad5d60fb1,21.0.0-3-g4be5c26+3ad5d60fb1,21.0.0-3-g65f322c+e0b24896a3,21.0.0-3-g7d9da8d+616205b9df,21.0.0-3-ge02ed75+a9c1acf22d,21.0.0-4-g591bb35+a9c1acf22d,21.0.0-4-g65b4814+b60e2d390c,21.0.0-4-gccdca77+0de219a2bc,21.0.0-4-ge8a399c+6c55c39e83,21.0.0-5-gd00fb1e+05fce91b99,21.0.0-6-gc675373+3ad5d60fb1,21.0.0-64-g1122c245+4fb2b8f86e,21.0.0-7-g04766d7+cd19d05db2,21.0.0-7-gdf92d54+04719a4bac,21.0.0-8-g5674e7b+d1bd76f71f,master-gac4afde19b+a9c1acf22d,w.2021.13
LSST Data Management Base Package
Public Member Functions | Public Attributes | Static Public Attributes | List of all members
lsst.pipe.tasks.ingestCalibs.IngestCalibsTask Class Reference
Inheritance diagram for lsst.pipe.tasks.ingestCalibs.IngestCalibsTask:
lsst.pipe.tasks.ingest.IngestTask lsst.pipe.base.task.Task lsst.pipe.tasks.ingestCuratedCalibs.IngestCuratedCalibsTask

Public Member Functions

def run (self, args)
 
def parseAndRun (cls)
 
def prepareTask (cls, root=None, dryrun=False, mode="move", create=False, ignoreIngested=False)
 
def ingest (self, infile, outfile, mode="move", dryrun=False)
 
def isBadFile (self, filename, badFileList)
 
def isBadId (self, info, badIdList)
 
def expandFiles (self, fileNameList)
 Expand a set of filenames and globs, returning a list of filenames. More...
 
def runFile (self, infile, registry, args, pos)
 Examine and ingest a single file. More...
 
def ingestFiles (self, fileList)
 
def emptyMetadata (self)
 
def getSchemaCatalogs (self)
 
def getAllSchemaCatalogs (self)
 
def getFullMetadata (self)
 
def getFullName (self)
 
def getName (self)
 
def getTaskDict (self)
 
def makeSubtask (self, name, **keyArgs)
 
def timer (self, name, logLevel=Log.DEBUG)
 
def makeField (cls, doc)
 
def __reduce__ (self)
 

Public Attributes

 metadata
 
 log
 
 config
 

Static Public Attributes

 ConfigClass = IngestCalibsConfig
 
 ArgumentParser = IngestCalibsArgumentParser
 

Detailed Description

Task that generates registry for calibration images

Definition at line 231 of file ingestCalibs.py.

Member Function Documentation

◆ __reduce__()

def lsst.pipe.base.task.Task.__reduce__ (   self)
inherited
Pickler.

Reimplemented in lsst.pipe.drivers.multiBandDriver.MultiBandDriverTask, and lsst.pipe.drivers.coaddDriver.CoaddDriverTask.

Definition at line 432 of file task.py.

432  def __reduce__(self):
433  """Pickler.
434  """
435  return self._unpickle_via_factory, (self.__class__, [], self._reduce_kwargs())

◆ emptyMetadata()

def lsst.pipe.base.task.Task.emptyMetadata (   self)
inherited
Empty (clear) the metadata for this Task and all sub-Tasks.

Definition at line 166 of file task.py.

166  def emptyMetadata(self):
167  """Empty (clear) the metadata for this Task and all sub-Tasks.
168  """
169  for subtask in self._taskDict.values():
170  subtask.metadata = dafBase.PropertyList()
171 
Class for storing ordered metadata with comments.
Definition: PropertyList.h:68

◆ expandFiles()

def lsst.pipe.tasks.ingest.IngestTask.expandFiles (   self,
  fileNameList 
)
inherited

Expand a set of filenames and globs, returning a list of filenames.

    @param fileNameList A list of files and glob patterns

    N.b. globs obey Posix semantics, so a pattern that matches nothing is returned unchanged

Definition at line 557 of file ingest.py.

557  def expandFiles(self, fileNameList):
558  """!Expand a set of filenames and globs, returning a list of filenames
559 
560  @param fileNameList A list of files and glob patterns
561 
562  N.b. globs obey Posix semantics, so a pattern that matches nothing is returned unchanged
563  """
564  filenameList = []
565  for globPattern in fileNameList:
566  files = glob(globPattern)
567 
568  if not files: # posix behaviour is to return pattern unchanged
569  self.log.warn("%s doesn't match any file" % globPattern)
570  continue
571 
572  filenameList.extend(files)
573 
574  return filenameList
575 

◆ getAllSchemaCatalogs()

def lsst.pipe.base.task.Task.getAllSchemaCatalogs (   self)
inherited
Get schema catalogs for all tasks in the hierarchy, combining the
results into a single dict.

Returns
-------
schemacatalogs : `dict`
    Keys are butler dataset type, values are a empty catalog (an
    instance of the appropriate `lsst.afw.table` Catalog type) for all
    tasks in the hierarchy, from the top-level task down
    through all subtasks.

Notes
-----
This method may be called on any task in the hierarchy; it will return
the same answer, regardless.

The default implementation should always suffice. If your subtask uses
schemas the override `Task.getSchemaCatalogs`, not this method.

Definition at line 204 of file task.py.

204  def getAllSchemaCatalogs(self):
205  """Get schema catalogs for all tasks in the hierarchy, combining the
206  results into a single dict.
207 
208  Returns
209  -------
210  schemacatalogs : `dict`
211  Keys are butler dataset type, values are a empty catalog (an
212  instance of the appropriate `lsst.afw.table` Catalog type) for all
213  tasks in the hierarchy, from the top-level task down
214  through all subtasks.
215 
216  Notes
217  -----
218  This method may be called on any task in the hierarchy; it will return
219  the same answer, regardless.
220 
221  The default implementation should always suffice. If your subtask uses
222  schemas the override `Task.getSchemaCatalogs`, not this method.
223  """
224  schemaDict = self.getSchemaCatalogs()
225  for subtask in self._taskDict.values():
226  schemaDict.update(subtask.getSchemaCatalogs())
227  return schemaDict
228 

◆ getFullMetadata()

def lsst.pipe.base.task.Task.getFullMetadata (   self)
inherited
Get metadata for all tasks.

Returns
-------
metadata : `lsst.daf.base.PropertySet`
    The `~lsst.daf.base.PropertySet` keys are the full task name.
    Values are metadata for the top-level task and all subtasks,
    sub-subtasks, etc.

Notes
-----
The returned metadata includes timing information (if
``@timer.timeMethod`` is used) and any metadata set by the task. The
name of each item consists of the full task name with ``.`` replaced
by ``:``, followed by ``.`` and the name of the item, e.g.::

    topLevelTaskName:subtaskName:subsubtaskName.itemName

using ``:`` in the full task name disambiguates the rare situation
that a task has a subtask and a metadata item with the same name.

Definition at line 229 of file task.py.

229  def getFullMetadata(self):
230  """Get metadata for all tasks.
231 
232  Returns
233  -------
234  metadata : `lsst.daf.base.PropertySet`
235  The `~lsst.daf.base.PropertySet` keys are the full task name.
236  Values are metadata for the top-level task and all subtasks,
237  sub-subtasks, etc.
238 
239  Notes
240  -----
241  The returned metadata includes timing information (if
242  ``@timer.timeMethod`` is used) and any metadata set by the task. The
243  name of each item consists of the full task name with ``.`` replaced
244  by ``:``, followed by ``.`` and the name of the item, e.g.::
245 
246  topLevelTaskName:subtaskName:subsubtaskName.itemName
247 
248  using ``:`` in the full task name disambiguates the rare situation
249  that a task has a subtask and a metadata item with the same name.
250  """
251  fullMetadata = dafBase.PropertySet()
252  for fullName, task in self.getTaskDict().items():
253  fullMetadata.set(fullName.replace(".", ":"), task.metadata)
254  return fullMetadata
255 
std::vector< SchemaItem< Flag > > * items
Class for storing generic metadata.
Definition: PropertySet.h:67

◆ getFullName()

def lsst.pipe.base.task.Task.getFullName (   self)
inherited
Get the task name as a hierarchical name including parent task
names.

Returns
-------
fullName : `str`
    The full name consists of the name of the parent task and each
    subtask separated by periods. For example:

    - The full name of top-level task "top" is simply "top".
    - The full name of subtask "sub" of top-level task "top" is
      "top.sub".
    - The full name of subtask "sub2" of subtask "sub" of top-level
      task "top" is "top.sub.sub2".

Definition at line 256 of file task.py.

256  def getFullName(self):
257  """Get the task name as a hierarchical name including parent task
258  names.
259 
260  Returns
261  -------
262  fullName : `str`
263  The full name consists of the name of the parent task and each
264  subtask separated by periods. For example:
265 
266  - The full name of top-level task "top" is simply "top".
267  - The full name of subtask "sub" of top-level task "top" is
268  "top.sub".
269  - The full name of subtask "sub2" of subtask "sub" of top-level
270  task "top" is "top.sub.sub2".
271  """
272  return self._fullName
273 

◆ getName()

def lsst.pipe.base.task.Task.getName (   self)
inherited
Get the name of the task.

Returns
-------
taskName : `str`
    Name of the task.

See also
--------
getFullName

Definition at line 274 of file task.py.

274  def getName(self):
275  """Get the name of the task.
276 
277  Returns
278  -------
279  taskName : `str`
280  Name of the task.
281 
282  See also
283  --------
284  getFullName
285  """
286  return self._name
287 
std::string const & getName() const noexcept
Return a filter's name.
Definition: Filter.h:78

◆ getSchemaCatalogs()

def lsst.pipe.base.task.Task.getSchemaCatalogs (   self)
inherited
Get the schemas generated by this task.

Returns
-------
schemaCatalogs : `dict`
    Keys are butler dataset type, values are an empty catalog (an
    instance of the appropriate `lsst.afw.table` Catalog type) for
    this task.

Notes
-----

.. warning::

   Subclasses that use schemas must override this method. The default
   implementation returns an empty dict.

This method may be called at any time after the Task is constructed,
which means that all task schemas should be computed at construction
time, *not* when data is actually processed. This reflects the
philosophy that the schema should not depend on the data.

Returning catalogs rather than just schemas allows us to save e.g.
slots for SourceCatalog as well.

See also
--------
Task.getAllSchemaCatalogs

Definition at line 172 of file task.py.

172  def getSchemaCatalogs(self):
173  """Get the schemas generated by this task.
174 
175  Returns
176  -------
177  schemaCatalogs : `dict`
178  Keys are butler dataset type, values are an empty catalog (an
179  instance of the appropriate `lsst.afw.table` Catalog type) for
180  this task.
181 
182  Notes
183  -----
184 
185  .. warning::
186 
187  Subclasses that use schemas must override this method. The default
188  implementation returns an empty dict.
189 
190  This method may be called at any time after the Task is constructed,
191  which means that all task schemas should be computed at construction
192  time, *not* when data is actually processed. This reflects the
193  philosophy that the schema should not depend on the data.
194 
195  Returning catalogs rather than just schemas allows us to save e.g.
196  slots for SourceCatalog as well.
197 
198  See also
199  --------
200  Task.getAllSchemaCatalogs
201  """
202  return {}
203 

◆ getTaskDict()

def lsst.pipe.base.task.Task.getTaskDict (   self)
inherited
Get a dictionary of all tasks as a shallow copy.

Returns
-------
taskDict : `dict`
    Dictionary containing full task name: task object for the top-level
    task and all subtasks, sub-subtasks, etc.

Definition at line 288 of file task.py.

288  def getTaskDict(self):
289  """Get a dictionary of all tasks as a shallow copy.
290 
291  Returns
292  -------
293  taskDict : `dict`
294  Dictionary containing full task name: task object for the top-level
295  task and all subtasks, sub-subtasks, etc.
296  """
297  return self._taskDict.copy()
298 
def getTaskDict(config, taskDict=None, baseName="")

◆ ingest()

def lsst.pipe.tasks.ingest.IngestTask.ingest (   self,
  infile,
  outfile,
  mode = "move",
  dryrun = False 
)
inherited
Ingest a file into the image repository.

@param infile  Name of input file
@param outfile Name of output file (file in repository)
@param mode    Mode of ingest (copy/link/move/skip)
@param dryrun  Only report what would occur?
@param Success boolean

Definition at line 478 of file ingest.py.

478  def ingest(self, infile, outfile, mode="move", dryrun=False):
479  """Ingest a file into the image repository.
480 
481  @param infile Name of input file
482  @param outfile Name of output file (file in repository)
483  @param mode Mode of ingest (copy/link/move/skip)
484  @param dryrun Only report what would occur?
485  @param Success boolean
486  """
487  if mode == "skip":
488  return True
489  if dryrun:
490  self.log.info("Would %s from %s to %s" % (mode, infile, outfile))
491  return True
492  try:
493  outdir = os.path.dirname(outfile)
494  if not os.path.isdir(outdir):
495  try:
496  os.makedirs(outdir)
497  except OSError as exc:
498  # Silently ignore mkdir failures due to race conditions
499  if not os.path.isdir(outdir):
500  raise RuntimeError(f"Failed to create directory {outdir}") from exc
501  if os.path.lexists(outfile):
502  if self.config.clobber:
503  os.unlink(outfile)
504  else:
505  raise RuntimeError("File %s already exists; consider --config clobber=True" % outfile)
506 
507  if mode == "copy":
508  assertCanCopy(infile, outfile)
509  shutil.copyfile(infile, outfile)
510  elif mode == "link":
511  if os.path.exists(outfile):
512  if os.path.samefile(infile, outfile):
513  self.log.debug("Already linked %s to %s: ignoring" % (infile, outfile))
514  else:
515  self.log.warn("%s already has a file at the target location (%s): ignoring "
516  "(set clobber=True to overwrite)" % (infile, outfile))
517  return False
518  os.symlink(os.path.abspath(infile), outfile)
519  elif mode == "move":
520  assertCanCopy(infile, outfile)
521  shutil.move(infile, outfile)
522  else:
523  raise AssertionError("Unknown mode: %s" % mode)
524  self.log.info("%s --<%s>--> %s" % (infile, mode, outfile))
525  except Exception as e:
526  self.log.warn("Failed to %s %s to %s: %s" % (mode, infile, outfile, e))
527  if not self.config.allowError:
528  raise RuntimeError(f"Failed to {mode} {infile} to {outfile}") from e
529  return False
530  return True
531 
def assertCanCopy(fromPath, toPath)
Definition: ingest.py:650

◆ ingestFiles()

def lsst.pipe.tasks.ingest.IngestTask.ingestFiles (   self,
  fileList 
)
inherited
Ingest specified file or list of files and add them to the registry.

This method can only be called if `prepareTask` was used.

Parameters
----------
fileList : `str` or `list` [`str`]
    Pathname or list of pathnames of files to ingest.

Definition at line 632 of file ingest.py.

632  def ingestFiles(self, fileList):
633  """Ingest specified file or list of files and add them to the registry.
634 
635  This method can only be called if `prepareTask` was used.
636 
637  Parameters
638  ----------
639  fileList : `str` or `list` [`str`]
640  Pathname or list of pathnames of files to ingest.
641  """
642  if not hasattr(self, "_args"):
643  raise RuntimeError("Task not created with prepareTask")
644  if isinstance(fileList, str):
645  fileList = [fileList]
646  self._args.files = fileList
647  self.run(self._args)
648 
649 

◆ isBadFile()

def lsst.pipe.tasks.ingest.IngestTask.isBadFile (   self,
  filename,
  badFileList 
)
inherited
Return whether the file qualifies as bad

We match against the list of bad file patterns.

Definition at line 532 of file ingest.py.

532  def isBadFile(self, filename, badFileList):
533  """Return whether the file qualifies as bad
534 
535  We match against the list of bad file patterns.
536  """
537  filename = os.path.basename(filename)
538  if not badFileList:
539  return False
540  for badFile in badFileList:
541  if fnmatch(filename, badFile):
542  return True
543  return False
544 

◆ isBadId()

def lsst.pipe.tasks.ingest.IngestTask.isBadId (   self,
  info,
  badIdList 
)
inherited
Return whether the file information qualifies as bad

We match against the list of bad data identifiers.

Definition at line 545 of file ingest.py.

545  def isBadId(self, info, badIdList):
546  """Return whether the file information qualifies as bad
547 
548  We match against the list of bad data identifiers.
549  """
550  if not badIdList:
551  return False
552  for badId in badIdList:
553  if all(info[key] == value for key, value in badId.items()):
554  return True
555  return False
556 
bool all(CoordinateExpr< N > const &expr) noexcept
Return true if all elements are true.

◆ makeField()

def lsst.pipe.base.task.Task.makeField (   cls,
  doc 
)
inherited
Make a `lsst.pex.config.ConfigurableField` for this task.

Parameters
----------
doc : `str`
    Help text for the field.

Returns
-------
configurableField : `lsst.pex.config.ConfigurableField`
    A `~ConfigurableField` for this task.

Examples
--------
Provides a convenient way to specify this task is a subtask of another
task.

Here is an example of use:

.. code-block:: python

    class OtherTaskConfig(lsst.pex.config.Config):
        aSubtask = ATaskClass.makeField("brief description of task")

Definition at line 359 of file task.py.

359  def makeField(cls, doc):
360  """Make a `lsst.pex.config.ConfigurableField` for this task.
361 
362  Parameters
363  ----------
364  doc : `str`
365  Help text for the field.
366 
367  Returns
368  -------
369  configurableField : `lsst.pex.config.ConfigurableField`
370  A `~ConfigurableField` for this task.
371 
372  Examples
373  --------
374  Provides a convenient way to specify this task is a subtask of another
375  task.
376 
377  Here is an example of use:
378 
379  .. code-block:: python
380 
381  class OtherTaskConfig(lsst.pex.config.Config):
382  aSubtask = ATaskClass.makeField("brief description of task")
383  """
384  return ConfigurableField(doc=doc, target=cls)
385 

◆ makeSubtask()

def lsst.pipe.base.task.Task.makeSubtask (   self,
  name,
**  keyArgs 
)
inherited
Create a subtask as a new instance as the ``name`` attribute of this
task.

Parameters
----------
name : `str`
    Brief name of the subtask.
keyArgs
    Extra keyword arguments used to construct the task. The following
    arguments are automatically provided and cannot be overridden:

    - "config".
    - "parentTask".

Notes
-----
The subtask must be defined by ``Task.config.name``, an instance of
`~lsst.pex.config.ConfigurableField` or
`~lsst.pex.config.RegistryField`.

Definition at line 299 of file task.py.

299  def makeSubtask(self, name, **keyArgs):
300  """Create a subtask as a new instance as the ``name`` attribute of this
301  task.
302 
303  Parameters
304  ----------
305  name : `str`
306  Brief name of the subtask.
307  keyArgs
308  Extra keyword arguments used to construct the task. The following
309  arguments are automatically provided and cannot be overridden:
310 
311  - "config".
312  - "parentTask".
313 
314  Notes
315  -----
316  The subtask must be defined by ``Task.config.name``, an instance of
317  `~lsst.pex.config.ConfigurableField` or
318  `~lsst.pex.config.RegistryField`.
319  """
320  taskField = getattr(self.config, name, None)
321  if taskField is None:
322  raise KeyError(f"{self.getFullName()}'s config does not have field {name!r}")
323  subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
324  setattr(self, name, subtask)
325 

◆ parseAndRun()

def lsst.pipe.tasks.ingest.IngestTask.parseAndRun (   cls)
inherited
Parse the command-line arguments and run the Task.

Definition at line 428 of file ingest.py.

428  def parseAndRun(cls):
429  """Parse the command-line arguments and run the Task."""
430  task, args = cls._parse()
431  task.run(args)
432 

◆ prepareTask()

def lsst.pipe.tasks.ingest.IngestTask.prepareTask (   cls,
  root = None,
  dryrun = False,
  mode = "move",
  create = False,
  ignoreIngested = False 
)
inherited
Prepare for running the task repeatedly with `ingestFiles`.

Saves the parsed arguments, including the Butler and log, as a
private instance variable.

Parameters
----------
root : `str`, optional
    Repository root pathname.  If None, run the Task using the
    command line arguments, ignoring all other arguments below.
dryrun : `bool`, optional
    If True, don't perform any action; log what would have happened.
mode : `str`, optional
    How files are delivered to their destination.  Default is "move",
    unlike the command-line default of "link".
create : `bool`, optional
    If True, create a new registry, clobbering any old one present.
ignoreIngested : `bool`, optional
    If True, do not complain if the file is already present in the
    registry (and do nothing else).

Returns
-------
task : `IngestTask`
    If `root` was provided, the IngestTask instance

Definition at line 434 of file ingest.py.

435  ignoreIngested=False):
436  """Prepare for running the task repeatedly with `ingestFiles`.
437 
438  Saves the parsed arguments, including the Butler and log, as a
439  private instance variable.
440 
441  Parameters
442  ----------
443  root : `str`, optional
444  Repository root pathname. If None, run the Task using the
445  command line arguments, ignoring all other arguments below.
446  dryrun : `bool`, optional
447  If True, don't perform any action; log what would have happened.
448  mode : `str`, optional
449  How files are delivered to their destination. Default is "move",
450  unlike the command-line default of "link".
451  create : `bool`, optional
452  If True, create a new registry, clobbering any old one present.
453  ignoreIngested : `bool`, optional
454  If True, do not complain if the file is already present in the
455  registry (and do nothing else).
456 
457  Returns
458  -------
459  task : `IngestTask`
460  If `root` was provided, the IngestTask instance
461  """
462  sys.argv = ["IngestTask"]
463  sys.argv.append(root)
464  if dryrun:
465  sys.argv.append("--dry-run")
466  sys.argv.append("--mode")
467  sys.argv.append(mode)
468  if create:
469  sys.argv.append("--create")
470  if ignoreIngested:
471  sys.argv.append("--ignore-ingested")
472  sys.argv.append("__fakefile__") # needed for parsing, not used
473 
474  task, args = cls._parse()
475  task._args = args
476  return task
477 

◆ run()

def lsst.pipe.tasks.ingestCalibs.IngestCalibsTask.run (   self,
  args 
)
Ingest all specified files and add them to the registry

Reimplemented from lsst.pipe.tasks.ingest.IngestTask.

Reimplemented in lsst.pipe.tasks.ingestCuratedCalibs.IngestCuratedCalibsTask.

Definition at line 237 of file ingestCalibs.py.

237  def run(self, args):
238  """Ingest all specified files and add them to the registry"""
239  calibRoot = args.calib if args.calib is not None else args.output
240  filenameList = self.expandFiles(args.files)
241  with self.register.openRegistry(calibRoot, create=args.create, dryrun=args.dryrun) as registry:
242  calibTypes = set()
243  for infile in filenameList:
244  fileInfo, hduInfoList = self.parse.getInfo(infile)
245  calibType = self.parse.getCalibType(infile)
246  if calibType not in self.register.config.tables:
247  self.log.warn(str("Skipped adding %s of observation type '%s' to registry "
248  "(must be one of %s)" %
249  (infile, calibType, ", ".join(self.register.config.tables))))
250  continue
251  calibTypes.add(calibType)
252  if args.mode != 'skip':
253  outfile = self.parse.getDestination(args.butler, fileInfo, infile)
254  ingested = self.ingest(infile, outfile, mode=args.mode, dryrun=args.dryrun)
255  if not ingested:
256  self.log.warn(str("Failed to ingest %s of observation type '%s'" %
257  (infile, calibType)))
258  continue
259  if self.register.check(registry, fileInfo, table=calibType):
260  if args.ignoreIngested:
261  continue
262 
263  self.log.warn("%s: already ingested: %s" % (infile, fileInfo))
264  for info in hduInfoList:
265  self.register.addRow(registry, info, dryrun=args.dryrun,
266  create=args.create, table=calibType)
267  if not args.dryrun:
268  self.register.updateValidityRanges(registry, args.validity, tables=calibTypes)
269  else:
270  self.log.info("Would update validity ranges here, but dryrun")
daf::base::PropertySet * set
Definition: fits.cc:912
def run(self, skyInfo, tempExpRefList, imageScalerList, weightList, altMaskList=None, mask=None, supplementaryData=None)

◆ runFile()

def lsst.pipe.tasks.ingest.IngestTask.runFile (   self,
  infile,
  registry,
  args,
  pos 
)
inherited

Examine and ingest a single file.

    @param infile: File to process
    @param registry: Registry into which to insert Butler metadata, or None
    @param args: Parsed command-line arguments
    @param pos: Position number of this file in the input list
    @return parsed information from FITS HDUs if registry is None; or None

Definition at line 576 of file ingest.py.

576  def runFile(self, infile, registry, args, pos):
577  """!Examine and ingest a single file
578 
579  @param infile: File to process
580  @param registry: Registry into which to insert Butler metadata, or None
581  @param args: Parsed command-line arguments
582  @param pos: Position number of this file in the input list
583  @return parsed information from FITS HDUs if registry is None; or None
584  """
585  if self.isBadFile(infile, args.badFile):
586  self.log.info("Skipping declared bad file %s" % infile)
587  return None
588  try:
589  fileInfo, hduInfoList = self.parse.getInfo(infile)
590  except Exception as e:
591  if not self.config.allowError:
592  raise RuntimeError(f"Error parsing {infile}") from e
593  self.log.warn("Error parsing %s (%s); skipping" % (infile, e))
594  return None
595  if self.isBadId(fileInfo, args.badId.idList):
596  self.log.info("Skipping declared bad file %s: %s" % (infile, fileInfo))
597  return None
598  if registry is not None and self.register.check(registry, fileInfo):
599  if args.ignoreIngested:
600  return None
601  self.log.warn("%s: already ingested: %s" % (infile, fileInfo))
602  outfile = self.parse.getDestination(args.butler, fileInfo, infile)
603  if not self.ingest(infile, outfile, mode=args.mode, dryrun=args.dryrun):
604  return None
605  if hduInfoList is None:
606  return None
607  if registry is None:
608  return hduInfoList
609  for info in hduInfoList:
610  try:
611  self.register.addRow(registry, info, dryrun=args.dryrun, create=args.create)
612  except Exception as exc:
613  raise IngestError(f"Failed to register file {infile}", infile, pos) from exc
614  return None # No further registration should be performed
615 

◆ timer()

def lsst.pipe.base.task.Task.timer (   self,
  name,
  logLevel = Log.DEBUG 
)
inherited
Context manager to log performance data for an arbitrary block of
code.

Parameters
----------
name : `str`
    Name of code being timed; data will be logged using item name:
    ``Start`` and ``End``.
logLevel
    A `lsst.log` level constant.

Examples
--------
Creating a timer context:

.. code-block:: python

    with self.timer("someCodeToTime"):
        pass  # code to time

See also
--------
timer.logInfo

Definition at line 327 of file task.py.

327  def timer(self, name, logLevel=Log.DEBUG):
328  """Context manager to log performance data for an arbitrary block of
329  code.
330 
331  Parameters
332  ----------
333  name : `str`
334  Name of code being timed; data will be logged using item name:
335  ``Start`` and ``End``.
336  logLevel
337  A `lsst.log` level constant.
338 
339  Examples
340  --------
341  Creating a timer context:
342 
343  .. code-block:: python
344 
345  with self.timer("someCodeToTime"):
346  pass # code to time
347 
348  See also
349  --------
350  timer.logInfo
351  """
352  logInfo(obj=self, prefix=name + "Start", logLevel=logLevel)
353  try:
354  yield
355  finally:
356  logInfo(obj=self, prefix=name + "End", logLevel=logLevel)
357 
def logInfo(obj, prefix, logLevel=Log.DEBUG)
Definition: timer.py:63

Member Data Documentation

◆ ArgumentParser

lsst.pipe.tasks.ingestCalibs.IngestCalibsTask.ArgumentParser = IngestCalibsArgumentParser
static

Definition at line 234 of file ingestCalibs.py.

◆ config

lsst.pipe.base.task.Task.config
inherited

Definition at line 162 of file task.py.

◆ ConfigClass

lsst.pipe.tasks.ingestCalibs.IngestCalibsTask.ConfigClass = IngestCalibsConfig
static

Definition at line 233 of file ingestCalibs.py.

◆ log

lsst.pipe.base.task.Task.log
inherited

Definition at line 161 of file task.py.

◆ metadata

lsst.pipe.base.task.Task.metadata
inherited

Definition at line 134 of file task.py.


The documentation for this class was generated from the following file: