Inheritance diagram for lsst.pipe.tasks.ingestDefects.IngestDefectsTask:

Public Member Functions
def	run (self, args)

def	parseAndRun (cls)

def	ingest (self, infile, outfile, mode="move", dryrun=False)

def	isBadFile (self, filename, badFileList)

def	isBadId (self, info, badIdList)

def	expandFiles (self, fileNameList)
	Expand a set of filenames and globs, returning a list of filenames. More...

def	runFile (self, infile, registry, args)
	Examine and ingest a single file. More...

def	emptyMetadata (self)

def	getSchemaCatalogs (self)

def	getAllSchemaCatalogs (self)

def	getFullMetadata (self)

def	getFullName (self)

def	getName (self)

def	getTaskDict (self)

def	makeSubtask (self, name, keyArgs)

def	timer (self, name, logLevel=Log.DEBUG)

def	makeField (cls, doc)

def	__reduce__ (self)

Public Attributes
	metadata

	log

	config

Static Public Attributes
	ArgumentParser

	ConfigClass

Detailed Description

Task that generates registry for calibration images

Definition at line 29 of file ingestDefects.py.

Member Function Documentation

◆ reduce()

def lsst.pipe.base.task.Task.__reduce__ ( self )

inherited

Pickler.

Definition at line 373 of file task.py.

     def __reduce__(self):
         """Pickler.
         """
         return self.__class__, (self.config, self._name, self._parentTask, None)
 

◆ emptyMetadata()

def lsst.pipe.base.task.Task.emptyMetadata ( self )

inherited

Empty (clear) the metadata for this Task and all sub-Tasks.

Definition at line 153 of file task.py.

     def emptyMetadata(self):
         """Empty (clear) the metadata for this Task and all sub-Tasks.
         """
         for subtask in self._taskDict.values():
             subtask.metadata = dafBase.PropertyList()
 

◆ expandFiles()

def lsst.pipe.tasks.ingest.IngestTask.expandFiles	(	self,
		fileNameList
	)

inherited

Expand a set of filenames and globs, returning a list of filenames.

Parameters

fileNameList A list of files and glob patterns

N.b. globs obey Posix semantics, so a pattern that matches nothing is returned unchanged

Definition at line 499 of file ingest.py.

     def expandFiles(self, fileNameList):
         """!Expand a set of filenames and globs, returning a list of filenames
 
         @param fileNameList A list of files and glob patterns
 
         N.b. globs obey Posix semantics, so a pattern that matches nothing is returned unchanged
         """
         filenameList = []
         for globPattern in fileNameList:
             files = glob(globPattern)
 
             if not files:               # posix behaviour is to return pattern unchanged
                 self.log.warn("%s doesn't match any file" % globPattern)
                 continue
 
             filenameList.extend(files)
 
         return filenameList
 

◆ getAllSchemaCatalogs()

def lsst.pipe.base.task.Task.getAllSchemaCatalogs ( self )

inherited

Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.

Returns
-------
schemacatalogs : `dict`
    Keys are butler dataset type, values are a empty catalog (an instance of the appropriate
    lsst.afw.table Catalog type) for all tasks in the hierarchy, from the top-level task down
    through all subtasks.

Notes
-----
This method may be called on any task in the hierarchy; it will return the same answer, regardless.

The default implementation should always suffice. If your subtask uses schemas the override
`Task.getSchemaCatalogs`, not this method.

Definition at line 188 of file task.py.

     def getAllSchemaCatalogs(self):
         """Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.
 
         Returns
         -------
         schemacatalogs : `dict`
             Keys are butler dataset type, values are a empty catalog (an instance of the appropriate
             lsst.afw.table Catalog type) for all tasks in the hierarchy, from the top-level task down
             through all subtasks.
 
         Notes
         -----
         This method may be called on any task in the hierarchy; it will return the same answer, regardless.
 
         The default implementation should always suffice. If your subtask uses schemas the override
         `Task.getSchemaCatalogs`, not this method.
         """
         schemaDict = self.getSchemaCatalogs()
         for subtask in self._taskDict.values():
             schemaDict.update(subtask.getSchemaCatalogs())
         return schemaDict
 

◆ getFullMetadata()

def lsst.pipe.base.task.Task.getFullMetadata ( self )

inherited

Get metadata for all tasks.

Returns
-------
metadata : `lsst.daf.base.PropertySet`
    The `~lsst.daf.base.PropertySet` keys are the full task name. Values are metadata
    for the top-level task and all subtasks, sub-subtasks, etc..

Notes
-----
The returned metadata includes timing information (if ``@timer.timeMethod`` is used)
and any metadata set by the task. The name of each item consists of the full task name
with ``.`` replaced by ``:``, followed by ``.`` and the name of the item, e.g.::

    topLevelTaskName:subtaskName:subsubtaskName.itemName

using ``:`` in the full task name disambiguates the rare situation that a task has a subtask
and a metadata item with the same name.

Definition at line 210 of file task.py.

     def getFullMetadata(self):
         """Get metadata for all tasks.
 
         Returns
         -------
         metadata : `lsst.daf.base.PropertySet`
             The `~lsst.daf.base.PropertySet` keys are the full task name. Values are metadata
             for the top-level task and all subtasks, sub-subtasks, etc..
 
         Notes
         -----
         The returned metadata includes timing information (if ``@timer.timeMethod`` is used)
         and any metadata set by the task. The name of each item consists of the full task name
         with ``.`` replaced by ``:``, followed by ``.`` and the name of the item, e.g.::
 
             topLevelTaskName:subtaskName:subsubtaskName.itemName
 
         using ``:`` in the full task name disambiguates the rare situation that a task has a subtask
         and a metadata item with the same name.
         """
         fullMetadata = dafBase.PropertySet()
         for fullName, task in self.getTaskDict().items():
             fullMetadata.set(fullName.replace(".", ":"), task.metadata)
         return fullMetadata
 

◆ getFullName()

def lsst.pipe.base.task.Task.getFullName ( self )

inherited

Get the task name as a hierarchical name including parent task names.

Returns
-------
fullName : `str`
    The full name consists of the name of the parent task and each subtask separated by periods.
    For example:

    - The full name of top-level task "top" is simply "top".
    - The full name of subtask "sub" of top-level task "top" is "top.sub".
    - The full name of subtask "sub2" of subtask "sub" of top-level task "top" is "top.sub.sub2".

Definition at line 235 of file task.py.

     def getFullName(self):
         """Get the task name as a hierarchical name including parent task names.
 
         Returns
         -------
         fullName : `str`
             The full name consists of the name of the parent task and each subtask separated by periods.
             For example:
 
             - The full name of top-level task "top" is simply "top".
             - The full name of subtask "sub" of top-level task "top" is "top.sub".
             - The full name of subtask "sub2" of subtask "sub" of top-level task "top" is "top.sub.sub2".
         """
         return self._fullName
 

◆ getName()

def lsst.pipe.base.task.Task.getName ( self )

inherited

Get the name of the task.

Returns
-------
taskName : `str`
    Name of the task.

See also
--------
getFullName

Definition at line 250 of file task.py.

     def getName(self):
         """Get the name of the task.
 
         Returns
         -------
         taskName : `str`
             Name of the task.
 
         See also
         --------
         getFullName
         """
         return self._name
 

◆ getSchemaCatalogs()

def lsst.pipe.base.task.Task.getSchemaCatalogs ( self )

inherited

Get the schemas generated by this task.

Returns
-------
schemaCatalogs : `dict`
    Keys are butler dataset type, values are an empty catalog (an instance of the appropriate
    `lsst.afw.table` Catalog type) for this task.

Notes
-----

.. warning::

   Subclasses that use schemas must override this method. The default implemenation returns
   an empty dict.

This method may be called at any time after the Task is constructed, which means that all task
schemas should be computed at construction time, *not* when data is actually processed. This
reflects the philosophy that the schema should not depend on the data.

Returning catalogs rather than just schemas allows us to save e.g. slots for SourceCatalog as well.

See also
--------
Task.getAllSchemaCatalogs

Definition at line 159 of file task.py.

     def getSchemaCatalogs(self):
         """Get the schemas generated by this task.
 
         Returns
         -------
         schemaCatalogs : `dict`
             Keys are butler dataset type, values are an empty catalog (an instance of the appropriate
             `lsst.afw.table` Catalog type) for this task.
 
         Notes
         -----
 
         .. warning::
 
            Subclasses that use schemas must override this method. The default implemenation returns
            an empty dict.
 
         This method may be called at any time after the Task is constructed, which means that all task
         schemas should be computed at construction time, *not* when data is actually processed. This
         reflects the philosophy that the schema should not depend on the data.
 
         Returning catalogs rather than just schemas allows us to save e.g. slots for SourceCatalog as well.
 
         See also
         --------
         Task.getAllSchemaCatalogs
         """
         return {}
 

◆ getTaskDict()

def lsst.pipe.base.task.Task.getTaskDict ( self )

inherited

Get a dictionary of all tasks as a shallow copy.

Returns
-------
taskDict : `dict`
    Dictionary containing full task name: task object for the top-level task and all subtasks,
    sub-subtasks, etc..

Definition at line 264 of file task.py.

     def getTaskDict(self):
         """Get a dictionary of all tasks as a shallow copy.
 
         Returns
         -------
         taskDict : `dict`
             Dictionary containing full task name: task object for the top-level task and all subtasks,
             sub-subtasks, etc..
         """
         return self._taskDict.copy()
 

◆ ingest()

def lsst.pipe.tasks.ingest.IngestTask.ingest	(	self,
		infile,
		outfile,
		mode = `"move"`,
		dryrun = `False`
	)

inherited

Ingest a file into the image repository.

@param infile  Name of input file
@param outfile Name of output file (file in repository)
@param mode    Mode of ingest (copy/link/move/skip)
@param dryrun  Only report what would occur?
@param Success boolean

Definition at line 427 of file ingest.py.

     def ingest(self, infile, outfile, mode="move", dryrun=False):
         """Ingest a file into the image repository.
 
         @param infile  Name of input file
         @param outfile Name of output file (file in repository)
         @param mode    Mode of ingest (copy/link/move/skip)
         @param dryrun  Only report what would occur?
         @param Success boolean
         """
         if mode == "skip":
             return True
         if dryrun:
             self.log.info("Would %s from %s to %s" % (mode, infile, outfile))
             return True
         try:
             outdir = os.path.dirname(outfile)
             if not os.path.isdir(outdir):
                 try:
                     os.makedirs(outdir)
                 except OSError:
                     # Silently ignore mkdir failures due to race conditions
                     if not os.path.isdir(outdir):
                         raise
             if os.path.lexists(outfile):
                 if self.config.clobber:
                     os.unlink(outfile)
                 else:
                     raise RuntimeError("File %s already exists; consider --config clobber=True" % outfile)
 
             if mode == "copy":
                 assertCanCopy(infile, outfile)
                 shutil.copyfile(infile, outfile)
             elif mode == "link":
                 os.symlink(os.path.abspath(infile), outfile)
             elif mode == "move":
                 assertCanCopy(infile, outfile)
                 shutil.move(infile, outfile)
             else:
                 raise AssertionError("Unknown mode: %s" % mode)
             self.log.info("%s --<%s>--> %s" % (infile, mode, outfile))
         except Exception as e:
             self.log.warn("Failed to %s %s to %s: %s" % (mode, infile, outfile, e))
             if not self.config.allowError:
                 raise
             return False
         return True
 

◆ isBadFile()

def lsst.pipe.tasks.ingest.IngestTask.isBadFile	(	self,
		filename,
		badFileList
	)

inherited

Return whether the file qualifies as bad

We match against the list of bad file patterns.

Definition at line 474 of file ingest.py.

     def isBadFile(self, filename, badFileList):
         """Return whether the file qualifies as bad
 
         We match against the list of bad file patterns.
         """
         filename = os.path.basename(filename)
         if not badFileList:
             return False
         for badFile in badFileList:
             if fnmatch(filename, badFile):
                 return True
         return False
 

◆ isBadId()

def lsst.pipe.tasks.ingest.IngestTask.isBadId	(	self,
		info,
		badIdList
	)

inherited

Return whether the file information qualifies as bad

We match against the list of bad data identifiers.

Definition at line 487 of file ingest.py.

     def isBadId(self, info, badIdList):
         """Return whether the file information qualifies as bad
 
         We match against the list of bad data identifiers.
         """
         if not badIdList:
             return False
         for badId in badIdList:
             if all(info[key] == value for key, value in badId.items()):
                 return True
         return False
 

◆ makeField()

def lsst.pipe.base.task.Task.makeField	(	cls,
		doc
	)

inherited

Make a `lsst.pex.config.ConfigurableField` for this task.

Parameters
----------
doc : `str`
    Help text for the field.

Returns
-------
configurableField : `lsst.pex.config.ConfigurableField`
    A `~ConfigurableField` for this task.

Examples
--------
Provides a convenient way to specify this task is a subtask of another task.

Here is an example of use::

    class OtherTaskConfig(lsst.pex.config.Config)
aSubtask = ATaskClass.makeField("a brief description of what this task does")

Definition at line 329 of file task.py.

     def makeField(cls, doc):
         """Make a `lsst.pex.config.ConfigurableField` for this task.
 
         Parameters
         ----------
         doc : `str`
             Help text for the field.
 
         Returns
         -------
         configurableField : `lsst.pex.config.ConfigurableField`
             A `~ConfigurableField` for this task.
 
         Examples
         --------
         Provides a convenient way to specify this task is a subtask of another task.
 
         Here is an example of use::
 
             class OtherTaskConfig(lsst.pex.config.Config)
                 aSubtask = ATaskClass.makeField("a brief description of what this task does")
         """
         return ConfigurableField(doc=doc, target=cls)
 

◆ makeSubtask()

def lsst.pipe.base.task.Task.makeSubtask	(	self,
		name,
		keyArgs
	)

inherited

Create a subtask as a new instance as the ``name`` attribute of this task.

Parameters
----------
name : `str`
    Brief name of the subtask.
keyArgs
    Extra keyword arguments used to construct the task. The following arguments are automatically
    provided and cannot be overridden:

    - "config".
    - "parentTask".

Notes
-----
The subtask must be defined by ``Task.config.name``, an instance of pex_config ConfigurableField
or RegistryField.

Definition at line 275 of file task.py.

     def makeSubtask(self, name, **keyArgs):
         """Create a subtask as a new instance as the ``name`` attribute of this task.
 
         Parameters
         ----------
         name : `str`
             Brief name of the subtask.
         keyArgs
             Extra keyword arguments used to construct the task. The following arguments are automatically
             provided and cannot be overridden:
 
             - "config".
             - "parentTask".
 
         Notes
         -----
         The subtask must be defined by ``Task.config.name``, an instance of pex_config ConfigurableField
         or RegistryField.
         """
         taskField = getattr(self.config, name, None)
         if taskField is None:
             raise KeyError("%s's config does not have field %r" % (self.getFullName(), name))
         subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
         setattr(self, name, subtask)
 

◆ parseAndRun()

def lsst.pipe.tasks.ingest.IngestTask.parseAndRun ( cls )

inherited

Parse the command-line arguments and run the Task

Definition at line 419 of file ingest.py.

     def parseAndRun(cls):
         """Parse the command-line arguments and run the Task"""
         config = cls.ConfigClass()
         parser = cls.ArgumentParser(name=cls._DefaultName)
         args = parser.parse_args(config)
         task = cls(config=args.config)
         task.run(args)
 

◆ run()

def lsst.pipe.tasks.ingestDefects.IngestDefectsTask.run	(	self,
		args
	)

Ingest all defect files and add them to the registry

Definition at line 35 of file ingestDefects.py.

     def run(self, args):
         """Ingest all defect files and add them to the registry"""
 
         try:
             camera = args.butler.get('camera')
             temp_dir = tempfile.mkdtemp()
             defects = read_all_defects(args.root, camera)
             file_names = []
             for d in defects:
                 for s in defects[d]:
                     file_name = f'defects_{d}_{s.isoformat()}.fits'
                     full_file_name = os.path.join(temp_dir, file_name)
                     self.log.info('%i defects written for sensor: %s and calibDate: %s' %
                                   (len(defects[d][s]), d, s.isoformat()))
                     defects[d][s].writeFits(full_file_name)
                     file_names.append(full_file_name)
             args.files = file_names
             args.mode = 'move'
             args.validity = None  # Validity range is determined from the files
             IngestCalibsTask.run(self, args)
         finally:
             shutil.rmtree(temp_dir)

◆ runFile()

def lsst.pipe.tasks.ingest.IngestTask.runFile	(	self,
		infile,
		registry,
		args
	)

inherited

Examine and ingest a single file.

Parameters

infile	File to process
args	Parsed command-line arguments

Returns: parsed information from FITS HDUs or None

Definition at line 518 of file ingest.py.

     def runFile(self, infile, registry, args):
         """!Examine and ingest a single file
 
         @param infile: File to process
         @param args: Parsed command-line arguments
         @return parsed information from FITS HDUs or None
         """
         if self.isBadFile(infile, args.badFile):
             self.log.info("Skipping declared bad file %s" % infile)
             return None
         try:
             fileInfo, hduInfoList = self.parse.getInfo(infile)
         except Exception as e:
             if not self.config.allowError:
                 raise
             self.log.warn("Error parsing %s (%s); skipping" % (infile, e))
             return None
         if self.isBadId(fileInfo, args.badId.idList):
             self.log.info("Skipping declared bad file %s: %s" % (infile, fileInfo))
             return
         if registry is not None and self.register.check(registry, fileInfo):
             if args.ignoreIngested:
                 return None
             self.log.warn("%s: already ingested: %s" % (infile, fileInfo))
         outfile = self.parse.getDestination(args.butler, fileInfo, infile)
         if not self.ingest(infile, outfile, mode=args.mode, dryrun=args.dryrun):
             return None
         return hduInfoList
 

◆ timer()

def lsst.pipe.base.task.Task.timer	(	self,
		name,
		logLevel = `Log.DEBUG`
	)

inherited

Context manager to log performance data for an arbitrary block of code.

Parameters
----------
name : `str`
    Name of code being timed; data will be logged using item name: ``Start`` and ``End``.
logLevel
    A `lsst.log` level constant.

Examples
--------
Creating a timer context::

    with self.timer("someCodeToTime"):
pass  # code to time

See also
--------
timer.logInfo

Definition at line 301 of file task.py.

     def timer(self, name, logLevel=Log.DEBUG):
         """Context manager to log performance data for an arbitrary block of code.
 
         Parameters
         ----------
         name : `str`
             Name of code being timed; data will be logged using item name: ``Start`` and ``End``.
         logLevel
             A `lsst.log` level constant.
 
         Examples
         --------
         Creating a timer context::
 
             with self.timer("someCodeToTime"):
                 pass  # code to time
 
         See also
         --------
         timer.logInfo
         """
         logInfo(obj=self, prefix=name + "Start", logLevel=logLevel)
         try:
             yield
         finally:
             logInfo(obj=self, prefix=name + "End", logLevel=logLevel)
 

Member Data Documentation

◆ ArgumentParser

lsst.pipe.tasks.ingestDefects.IngestDefectsTask.ArgumentParser

static

Definition at line 31 of file ingestDefects.py.

◆ config

lsst.pipe.base.task.Task.config

inherited

Definition at line 149 of file task.py.

◆ ConfigClass

lsst.pipe.tasks.ingestDefects.IngestDefectsTask.ConfigClass

static

Definition at line 33 of file ingestDefects.py.

◆ log

lsst.pipe.base.task.Task.log

inherited

Definition at line 148 of file task.py.

◆ metadata

lsst.pipe.base.task.Task.metadata

inherited

Definition at line 121 of file task.py.

The documentation for this class was generated from the following file:

/j/snowflake/release/lsstsw/stack/Linux64/pipe_tasks/19.0.0+3/python/lsst/pipe/tasks/ingestDefects.py

Public Member Functions

Public Attributes

Static Public Attributes

Detailed Description

Member Function Documentation

◆ __reduce__()

◆ emptyMetadata()

◆ expandFiles()

◆ getAllSchemaCatalogs()

◆ getFullMetadata()

◆ getFullName()

◆ getName()

◆ getSchemaCatalogs()

◆ getTaskDict()

◆ ingest()

◆ isBadFile()

◆ isBadId()

◆ makeField()

◆ makeSubtask()

◆ parseAndRun()

◆ run()

◆ runFile()

◆ timer()

Member Data Documentation

◆ ArgumentParser

◆ config

◆ ConfigClass

◆ log

◆ metadata

◆ reduce()