Inheritance diagram for lsst.pipe.tasks.ingest.ParseTask:

Public Member Functions
def	getInfo (self, filename)

def	getInfoFromMetadata (self, md, info=None)

def	translate_date (self, md)

def	translate_filter (self, md)

def	getDestination (self, butler, info, filename)

def	emptyMetadata (self)

def	getSchemaCatalogs (self)

def	getAllSchemaCatalogs (self)

def	getFullMetadata (self)

def	getFullName (self)

def	getName (self)

def	getTaskDict (self)

def	makeSubtask (self, name, keyArgs)

def	timer (self, name, logLevel=Log.DEBUG)

def	makeField (cls, doc)

def	__reduce__ (self)

Static Public Member Functions
def	getExtensionName (md)

Public Attributes
	metadata

	log

	config

Static Public Attributes
	ConfigClass = ParseConfig

Detailed Description

Task that will parse the filename and/or its contents to get the required information
for putting the file in the correct location and populating the registry.

Definition at line 67 of file ingest.py.

Member Function Documentation

◆ reduce()

def lsst.pipe.base.task.Task.__reduce__ ( self )

inherited

Pickler.

Definition at line 373 of file task.py.

     def __reduce__(self):
         """Pickler.
         """
         return self.__class__, (self.config, self._name, self._parentTask, None)
 

◆ emptyMetadata()

def lsst.pipe.base.task.Task.emptyMetadata ( self )

inherited

Empty (clear) the metadata for this Task and all sub-Tasks.

Definition at line 153 of file task.py.

     def emptyMetadata(self):
         """Empty (clear) the metadata for this Task and all sub-Tasks.
         """
         for subtask in self._taskDict.values():
             subtask.metadata = dafBase.PropertyList()
 

◆ getAllSchemaCatalogs()

def lsst.pipe.base.task.Task.getAllSchemaCatalogs ( self )

inherited

Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.

Returns
-------
schemacatalogs : `dict`
    Keys are butler dataset type, values are a empty catalog (an instance of the appropriate
    lsst.afw.table Catalog type) for all tasks in the hierarchy, from the top-level task down
    through all subtasks.

Notes
-----
This method may be called on any task in the hierarchy; it will return the same answer, regardless.

The default implementation should always suffice. If your subtask uses schemas the override
`Task.getSchemaCatalogs`, not this method.

Definition at line 188 of file task.py.

     def getAllSchemaCatalogs(self):
         """Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.
 
         Returns
         -------
         schemacatalogs : `dict`
             Keys are butler dataset type, values are a empty catalog (an instance of the appropriate
             lsst.afw.table Catalog type) for all tasks in the hierarchy, from the top-level task down
             through all subtasks.
 
         Notes
         -----
         This method may be called on any task in the hierarchy; it will return the same answer, regardless.
 
         The default implementation should always suffice. If your subtask uses schemas the override
         `Task.getSchemaCatalogs`, not this method.
         """
         schemaDict = self.getSchemaCatalogs()
         for subtask in self._taskDict.values():
             schemaDict.update(subtask.getSchemaCatalogs())
         return schemaDict
 

◆ getDestination()

def lsst.pipe.tasks.ingest.ParseTask.getDestination	(	self,
		butler,
		info,
		filename
	)

Get destination for the file

@param butler      Data butler
@param info        File properties, used as dataId for the butler
@param filename    Input filename
@return Destination filename

Definition at line 181 of file ingest.py.

     def getDestination(self, butler, info, filename):
         """Get destination for the file
 
         @param butler      Data butler
         @param info        File properties, used as dataId for the butler
         @param filename    Input filename
         @return Destination filename
         """
         raw = butler.get("raw_filename", info)[0]
         # Ensure filename is devoid of cfitsio directions about HDUs
         c = raw.find("[")
         if c > 0:
             raw = raw[:c]
         return raw
 
 

◆ getExtensionName()

def lsst.pipe.tasks.ingest.ParseTask.getExtensionName ( md )

static

Get the name of an extension.
@param md: PropertySet like one obtained from lsst.afw.fits.readMetadata)
@return Name of the extension if it exists.  None otherwise.

Definition at line 107 of file ingest.py.

     def getExtensionName(md):
         """ Get the name of an extension.
         @param md: PropertySet like one obtained from lsst.afw.fits.readMetadata)
         @return Name of the extension if it exists.  None otherwise.
         """
         try:
             # This returns a tuple
             ext = md.getScalar("EXTNAME")
             return ext[1]
         except lsst.pex.exceptions.Exception:
             return None
 

◆ getFullMetadata()

def lsst.pipe.base.task.Task.getFullMetadata ( self )

inherited

Get metadata for all tasks.

Returns
-------
metadata : `lsst.daf.base.PropertySet`
    The `~lsst.daf.base.PropertySet` keys are the full task name. Values are metadata
    for the top-level task and all subtasks, sub-subtasks, etc..

Notes
-----
The returned metadata includes timing information (if ``@timer.timeMethod`` is used)
and any metadata set by the task. The name of each item consists of the full task name
with ``.`` replaced by ``:``, followed by ``.`` and the name of the item, e.g.::

    topLevelTaskName:subtaskName:subsubtaskName.itemName

using ``:`` in the full task name disambiguates the rare situation that a task has a subtask
and a metadata item with the same name.

Definition at line 210 of file task.py.

     def getFullMetadata(self):
         """Get metadata for all tasks.
 
         Returns
         -------
         metadata : `lsst.daf.base.PropertySet`
             The `~lsst.daf.base.PropertySet` keys are the full task name. Values are metadata
             for the top-level task and all subtasks, sub-subtasks, etc..
 
         Notes
         -----
         The returned metadata includes timing information (if ``@timer.timeMethod`` is used)
         and any metadata set by the task. The name of each item consists of the full task name
         with ``.`` replaced by ``:``, followed by ``.`` and the name of the item, e.g.::
 
             topLevelTaskName:subtaskName:subsubtaskName.itemName
 
         using ``:`` in the full task name disambiguates the rare situation that a task has a subtask
         and a metadata item with the same name.
         """
         fullMetadata = dafBase.PropertySet()
         for fullName, task in self.getTaskDict().items():
             fullMetadata.set(fullName.replace(".", ":"), task.metadata)
         return fullMetadata
 

◆ getFullName()

def lsst.pipe.base.task.Task.getFullName ( self )

inherited

Get the task name as a hierarchical name including parent task names.

Returns
-------
fullName : `str`
    The full name consists of the name of the parent task and each subtask separated by periods.
    For example:

    - The full name of top-level task "top" is simply "top".
    - The full name of subtask "sub" of top-level task "top" is "top.sub".
    - The full name of subtask "sub2" of subtask "sub" of top-level task "top" is "top.sub.sub2".

Definition at line 235 of file task.py.

     def getFullName(self):
         """Get the task name as a hierarchical name including parent task names.
 
         Returns
         -------
         fullName : `str`
             The full name consists of the name of the parent task and each subtask separated by periods.
             For example:
 
             - The full name of top-level task "top" is simply "top".
             - The full name of subtask "sub" of top-level task "top" is "top.sub".
             - The full name of subtask "sub2" of subtask "sub" of top-level task "top" is "top.sub.sub2".
         """
         return self._fullName
 

◆ getInfo()

def lsst.pipe.tasks.ingest.ParseTask.getInfo	(	self,
		filename
	)

Get information about the image from the filename and its contents

Here, we open the image and parse the header, but one could also look at the filename itself
and derive information from that, or set values from the configuration.

@param filename    Name of file to inspect
@return File properties; list of file properties for each extension

Definition at line 72 of file ingest.py.

     def getInfo(self, filename):
         """Get information about the image from the filename and its contents
 
         Here, we open the image and parse the header, but one could also look at the filename itself
         and derive information from that, or set values from the configuration.
 
         @param filename    Name of file to inspect
         @return File properties; list of file properties for each extension
         """
         md = readMetadata(filename, self.config.hdu)
         phuInfo = self.getInfoFromMetadata(md)
         if len(self.config.extnames) == 0:
             # No extensions to worry about
             return phuInfo, [phuInfo]
         # Look in the provided extensions
         extnames = set(self.config.extnames)
         extnum = 0
         infoList = []
         while len(extnames) > 0:
             extnum += 1
             try:
                 md = readMetadata(filename, extnum)
             except Exception as e:
                 self.log.warn("Error reading %s extensions %s: %s" % (filename, extnames, e))
                 break
             ext = self.getExtensionName(md)
             if ext in extnames:
                 hduInfo = self.getInfoFromMetadata(md, info=phuInfo.copy())
                 # We need the HDU number when registering MEF files.
                 hduInfo["hdu"] = extnum
                 infoList.append(hduInfo)
                 extnames.discard(ext)
         return phuInfo, infoList
 

◆ getInfoFromMetadata()

def lsst.pipe.tasks.ingest.ParseTask.getInfoFromMetadata	(	self,
		md,
		info = `None`
	)

Attempt to pull the desired information out of the header

This is done through two mechanisms:
* translation: a property is set directly from the relevant header keyword
* translator: a property is set with the result of calling a method

The translator methods receive the header metadata and should return the
appropriate value, or None if the value cannot be determined.

@param md      FITS header
@param info    File properties, to be supplemented
@return info

Definition at line 119 of file ingest.py.

     def getInfoFromMetadata(self, md, info=None):
         """Attempt to pull the desired information out of the header
 
         This is done through two mechanisms:
         * translation: a property is set directly from the relevant header keyword
         * translator: a property is set with the result of calling a method
 
         The translator methods receive the header metadata and should return the
         appropriate value, or None if the value cannot be determined.
 
         @param md      FITS header
         @param info    File properties, to be supplemented
         @return info
         """
         if info is None:
             info = {}
         for p, h in self.config.translation.items():
             if md.exists(h):
                 value = md.getScalar(h)
                 if isinstance(value, str):
                     value = value.strip()
                 info[p] = value
             elif p in self.config.defaults:
                 info[p] = self.config.defaults[p]
             else:
                 self.log.warn("Unable to find value for %s (derived from %s)" % (p, h))
         for p, t in self.config.translators.items():
             func = getattr(self, t)
             try:
                 value = func(md)
             except Exception as e:
                 self.log.warn("%s failed to translate %s: %s", t, p, e)
                 value = None
             if value is not None:
                 info[p] = value
         return info
 

◆ getName()

def lsst.pipe.base.task.Task.getName ( self )

inherited

Get the name of the task.

Returns
-------
taskName : `str`
    Name of the task.

See also
--------
getFullName

Definition at line 250 of file task.py.

     def getName(self):
         """Get the name of the task.
 
         Returns
         -------
         taskName : `str`
             Name of the task.
 
         See also
         --------
         getFullName
         """
         return self._name
 

◆ getSchemaCatalogs()

def lsst.pipe.base.task.Task.getSchemaCatalogs ( self )

inherited

Get the schemas generated by this task.

Returns
-------
schemaCatalogs : `dict`
    Keys are butler dataset type, values are an empty catalog (an instance of the appropriate
    `lsst.afw.table` Catalog type) for this task.

Notes
-----

.. warning::

   Subclasses that use schemas must override this method. The default implemenation returns
   an empty dict.

This method may be called at any time after the Task is constructed, which means that all task
schemas should be computed at construction time, *not* when data is actually processed. This
reflects the philosophy that the schema should not depend on the data.

Returning catalogs rather than just schemas allows us to save e.g. slots for SourceCatalog as well.

See also
--------
Task.getAllSchemaCatalogs

Definition at line 159 of file task.py.

     def getSchemaCatalogs(self):
         """Get the schemas generated by this task.
 
         Returns
         -------
         schemaCatalogs : `dict`
             Keys are butler dataset type, values are an empty catalog (an instance of the appropriate
             `lsst.afw.table` Catalog type) for this task.
 
         Notes
         -----
 
         .. warning::
 
            Subclasses that use schemas must override this method. The default implemenation returns
            an empty dict.
 
         This method may be called at any time after the Task is constructed, which means that all task
         schemas should be computed at construction time, *not* when data is actually processed. This
         reflects the philosophy that the schema should not depend on the data.
 
         Returning catalogs rather than just schemas allows us to save e.g. slots for SourceCatalog as well.
 
         See also
         --------
         Task.getAllSchemaCatalogs
         """
         return {}
 

◆ getTaskDict()

def lsst.pipe.base.task.Task.getTaskDict ( self )

inherited

Get a dictionary of all tasks as a shallow copy.

Returns
-------
taskDict : `dict`
    Dictionary containing full task name: task object for the top-level task and all subtasks,
    sub-subtasks, etc..

Definition at line 264 of file task.py.

     def getTaskDict(self):
         """Get a dictionary of all tasks as a shallow copy.
 
         Returns
         -------
         taskDict : `dict`
             Dictionary containing full task name: task object for the top-level task and all subtasks,
             sub-subtasks, etc..
         """
         return self._taskDict.copy()
 

◆ makeField()

def lsst.pipe.base.task.Task.makeField	(	cls,
		doc
	)

inherited

Make a `lsst.pex.config.ConfigurableField` for this task.

Parameters
----------
doc : `str`
    Help text for the field.

Returns
-------
configurableField : `lsst.pex.config.ConfigurableField`
    A `~ConfigurableField` for this task.

Examples
--------
Provides a convenient way to specify this task is a subtask of another task.

Here is an example of use::

    class OtherTaskConfig(lsst.pex.config.Config)
aSubtask = ATaskClass.makeField("a brief description of what this task does")

Definition at line 329 of file task.py.

     def makeField(cls, doc):
         """Make a `lsst.pex.config.ConfigurableField` for this task.
 
         Parameters
         ----------
         doc : `str`
             Help text for the field.
 
         Returns
         -------
         configurableField : `lsst.pex.config.ConfigurableField`
             A `~ConfigurableField` for this task.
 
         Examples
         --------
         Provides a convenient way to specify this task is a subtask of another task.
 
         Here is an example of use::
 
             class OtherTaskConfig(lsst.pex.config.Config)
                 aSubtask = ATaskClass.makeField("a brief description of what this task does")
         """
         return ConfigurableField(doc=doc, target=cls)
 

◆ makeSubtask()

def lsst.pipe.base.task.Task.makeSubtask	(	self,
		name,
		keyArgs
	)

inherited

Create a subtask as a new instance as the ``name`` attribute of this task.

Parameters
----------
name : `str`
    Brief name of the subtask.
keyArgs
    Extra keyword arguments used to construct the task. The following arguments are automatically
    provided and cannot be overridden:

    - "config".
    - "parentTask".

Notes
-----
The subtask must be defined by ``Task.config.name``, an instance of pex_config ConfigurableField
or RegistryField.

Definition at line 275 of file task.py.

     def makeSubtask(self, name, **keyArgs):
         """Create a subtask as a new instance as the ``name`` attribute of this task.
 
         Parameters
         ----------
         name : `str`
             Brief name of the subtask.
         keyArgs
             Extra keyword arguments used to construct the task. The following arguments are automatically
             provided and cannot be overridden:
 
             - "config".
             - "parentTask".
 
         Notes
         -----
         The subtask must be defined by ``Task.config.name``, an instance of pex_config ConfigurableField
         or RegistryField.
         """
         taskField = getattr(self.config, name, None)
         if taskField is None:
             raise KeyError("%s's config does not have field %r" % (self.getFullName(), name))
         subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
         setattr(self, name, subtask)
 

◆ timer()

def lsst.pipe.base.task.Task.timer	(	self,
		name,
		logLevel = `Log.DEBUG`
	)

inherited

Context manager to log performance data for an arbitrary block of code.

Parameters
----------
name : `str`
    Name of code being timed; data will be logged using item name: ``Start`` and ``End``.
logLevel
    A `lsst.log` level constant.

Examples
--------
Creating a timer context::

    with self.timer("someCodeToTime"):
pass  # code to time

See also
--------
timer.logInfo

Definition at line 301 of file task.py.

     def timer(self, name, logLevel=Log.DEBUG):
         """Context manager to log performance data for an arbitrary block of code.
 
         Parameters
         ----------
         name : `str`
             Name of code being timed; data will be logged using item name: ``Start`` and ``End``.
         logLevel
             A `lsst.log` level constant.
 
         Examples
         --------
         Creating a timer context::
 
             with self.timer("someCodeToTime"):
                 pass  # code to time
 
         See also
         --------
         timer.logInfo
         """
         logInfo(obj=self, prefix=name + "Start", logLevel=logLevel)
         try:
             yield
         finally:
             logInfo(obj=self, prefix=name + "End", logLevel=logLevel)
 

◆ translate_date()

def lsst.pipe.tasks.ingest.ParseTask.translate_date	(	self,
		md
	)

Convert a full DATE-OBS to a mere date

Besides being an example of a translator, this is also generally useful.
It will only be used if listed as a translator in the configuration.

Definition at line 156 of file ingest.py.

     def translate_date(self, md):
         """Convert a full DATE-OBS to a mere date
 
         Besides being an example of a translator, this is also generally useful.
         It will only be used if listed as a translator in the configuration.
         """
         date = md.getScalar("DATE-OBS").strip()
         c = date.find("T")
         if c > 0:
             date = date[:c]
         return date
 

◆ translate_filter()

def lsst.pipe.tasks.ingest.ParseTask.translate_filter	(	self,
		md
	)

Translate a full filter description into a mere filter name

Besides being an example of a translator, this is also generally useful.
It will only be used if listed as a translator in the configuration.

Definition at line 168 of file ingest.py.

     def translate_filter(self, md):
         """Translate a full filter description into a mere filter name
 
         Besides being an example of a translator, this is also generally useful.
         It will only be used if listed as a translator in the configuration.
         """
         filterName = md.getScalar("FILTER").strip()
         filterName = filterName.strip()
         c = filterName.find(" ")
         if c > 0:
             filterName = filterName[:c]
         return filterName
 

Member Data Documentation

◆ config

lsst.pipe.base.task.Task.config

inherited

Definition at line 149 of file task.py.

◆ ConfigClass

lsst.pipe.tasks.ingest.ParseTask.ConfigClass = ParseConfig

static

Definition at line 70 of file ingest.py.

◆ log

lsst.pipe.base.task.Task.log

inherited

Definition at line 148 of file task.py.

◆ metadata

lsst.pipe.base.task.Task.metadata

inherited

Definition at line 121 of file task.py.

The documentation for this class was generated from the following file:

/j/snowflake/release/lsstsw/stack/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py

Public Member Functions

Static Public Member Functions

Public Attributes

Static Public Attributes

Detailed Description

Member Function Documentation

◆ __reduce__()

◆ emptyMetadata()

◆ getAllSchemaCatalogs()

◆ getDestination()

◆ getExtensionName()

◆ getFullMetadata()

◆ getFullName()

◆ getInfo()

◆ getInfoFromMetadata()

◆ getName()

◆ getSchemaCatalogs()

◆ getTaskDict()

◆ makeField()

◆ makeSubtask()

◆ timer()

◆ translate_date()

◆ translate_filter()

Member Data Documentation

◆ config

◆ ConfigClass

◆ log

◆ metadata

◆ reduce()